Replay Debugging of Complex Real-Time Systems: Experiences from Two Industrial Case Studies

摘要

Deterministic replay is a method for allowing complex multitasking real-timesystems to be debugged using standard interactive debuggers. Even thoughseveral replay techniques have been proposed for parallel, multi-tasking andreal-time systems, the solutions have so far lingered on a prototype academiclevel, with very little results to show from actual state-of-the-practicecommercial applications. This paper describes a major deterministic replaydebugging case study performed on a full-scale industrial robot control system,as well as a minor replay instrumentation case study performed on a militaryaircraft radar system. In this article, we will show that replay debugging isfeasible in complex multi-million lines of code software projects running ontop of off-the-shelf real-time operating systems. Furthermore, we will discusshow replay debugging can be introduced in existing systems withoutimpracticable analysis efforts. In addition, we will present benchmarkingresults from both studies, indicating that the instrumentation overhead isacceptable and affordable.